no there are no limits. But if you chose the ICMP monitor type and the servers don't respond to ping, it would show down. That's why I said it depends on the type of monitor you selected when adding servers to the pool.
You might also make sure you're on 2.0.1, there were many fixes made to relayd (the Load Balancer daemon we use) after 2.0